# ClojureVerse.org Maintenance & Upgrades - 27th October 2021
**URL:** https://clojureverse.org/t/clojureverse-org-maintenance-upgrades-27th-october-2021/8309
**Category:** Clojureverse Meta
**Created:** [October 27, 2021, 9:34am UTC](https://clojureverse.org/t/clojureverse-org-maintenance-upgrades-27th-october-2021/8309 "2021-10-27T09:34:18Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@oxalorg](https://clojureverse.org/u/oxalorg)
#### Post date: [October 27, 2021, 9:34am UTC](https://clojureverse.org/t/clojureverse-org-maintenance-upgrades-27th-october-2021/8309/1 "2021-10-27T09:34:18Z")
Hey folks,
Over the past few months [Clojureverse.org](http://Clojureverse.org) has been down a handful of times, the first time was mysterious (read here: [ClojureVerse Report: August 22 downtime / Planned changes to infra](https://v2.clojureverse.org/t/clojureverse-report-august-22-downtime-planned-changes-to-infra/8083)) but the next few times it was an easy fix to get the site online whenever it went down.
But it’s time to upgrade the infrastructure and bring things up-to date for more stability.
I’ll be spending some time today to do the following tasks
- move from DigitalOcean to Exoscale
- setup and install required software on the new server
- setup discourse on a staging subdomain and verify if it works
- change existing discourse instance into read-only LOCK mode
- export a fresh backup of the site
- import into the new instance
- verify if it works
- change DNS of [clojureverse.org](http://clojureverse.org) to new instance
- rebuild discourse with LetsEncrypt with changed domain settings
- disable read-only mode
Expect a bit of downtime today! 🙈 🔫
---
### Author:  [@pyr](https://clojureverse.org/u/pyr)
#### Post date: [November 12, 2021, 8:12am UTC](https://clojureverse.org/t/clojureverse-org-maintenance-upgrades-27th-october-2021/8309/2 "2021-11-12T08:12:20Z")
Hi @oxalorg,
Exoscale would be happy to help if you need anything, technically or materially!
---
### Author:  [@oxalorg](https://clojureverse.org/u/oxalorg)
#### Post date: [December 1, 2021, 11:24am UTC](https://clojureverse.org/t/clojureverse-org-maintenance-upgrades-27th-october-2021/8309/3 "2021-12-01T11:24:29Z")
Posting a [log](https://clojurians-log.clojureverse.org/clojureverse-ops/2021-10-27) of the things I did to get clojureverse up and running on a new server 😛
- [http://clojureverse.org](http://clojureverse.org) is now in READONLY mode while I’m creating a fresh backup
[oxalorg (Mitesh)](https://clojurians-log.clojureverse.org/_/_/users/U013MQC5YKD)[04:10:58](https://clojurians-log.clojureverse.org/clojureverse-ops/2021-10-27/1635396838.016300)
- [http://clojureverse.org](http://clojureverse.org) is now in READONLY mode while I’m creating a fresh backup
- WOW! Discourse isn’t importing it’s backups cleanly, their recommended solution is to manually delete entries and their links from postgres 🙈 (or to pay someone) Going to delete everything and try rebuilding it again
- Argh! Trying to rebuild it once again, and turns out the DNS changes didn’t propogate fully yet, and now hit the letsencrypt rate limit 🙈
- Fresh rebuilds didn’t work. I’ve reverted DNS back to old instance and after diving deep into whats wrong I found some issues in the data:
```auto
ERROR: could not create unique index "index_incoming_referers_on_path_and_incoming_domain_id"
DETAIL: Key (path, incoming_domain_id)=(/r/clojure.compact, 1229) is duplicated.
EXCEPTION: psql failed: DETAIL: Key (path, incoming_domain_id)=(/r/clojure.compact, 1229) is duplicated.
```
- Then I connected to our old instance pg container and tried to debug this
```auto
discourse=# select path, count(path) as cp from incoming_referers group by path, incoming_domain_id having COUNT(*) > 1;
path | cp
---------------------+----
/t:clojure | 2
/r/clojure.compact | 2
/r/clojure/.compact | 2
/r/clojure.compact | 2
(4 rows)
```
- Looks like there are duplicates where these shouldn’t exist
- okay I manually removed all conflicts from the postgres referrer table, these tables are related to link tables so I made sure to not break them:
```auto
discourse=# select * from incoming_referers where path LIKE '%/r/clojure/.compact%';
id | path | incoming_domain_id
------+---------------------+--------------------
1259 | /r/clojure/.compact | 829
8154 | /r/clojure/.compact | 829
(2 rows)
discourse=# update incoming_referers set path = '/r/clojure.compact/' where id=8154;
UPDATE 1
discourse=# select path, count(path) as cp from incoming_referers group by path, incoming_domain_id having COUNT(*) > 1;
```
- creating a new backup and trying to restore that now
- YES!! This worked 🎉 Going to try and work through the LetsEncrypt rate limit by manually syncing the certs from the previous instance 🙈
- Something wrong with SMTP configuration, looking into it!
- We’re done with the complete successful migration 😁 sharkdance clojure-spin! Emails are also working now but there are some background jobs running for the import process to complete. Due to this email notifications are disabled until all these tasks are completed. Will re-enable them soon!
- Processing of all background tasks are completed, I’ve enabled email notifications/digest! 🙂
- We’re done with the complete successful migration 😁 sharkdance clojure-spin! Emails are also working now but there are some background jobs running for the import process to complete. Due to this email notifications are disabled until all these tasks are completed. Will re-enable them soon!
---
### Author:  [@system](https://clojureverse.org/u/system)
#### Post date: [June 1, 2022, 11:25pm UTC](https://clojureverse.org/t/clojureverse-org-maintenance-upgrades-27th-october-2021/8309/4 "2022-06-01T23:25:24Z")
This topic was automatically closed 182 days after the last reply. New replies are no longer allowed.