1 Topic by fsantiago06111979

  • fsantiago06111979
  • Member
  • Offline
  • Registered: 2014-05-04
  • Posts: 229

Topic: backup / migration of existing spider archiver

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): 2023052301
- Deployed with iRedMail Easy or the downloadable installer? Easy
- Linux/BSD distribution name and version: Ubuntu 20.04.6 LTS
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): MySQL (mariadb)
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? No
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

no problem, just a question; do you have a user guide for proper backup / server migration of spider archiver? i currently run spider archiver v2.3.0 on Ubuntu 22.04.2 LTS.

Thanks.

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,793

Re: backup / migration of existing spider archiver

Hi Fabian,

All data used by Spider is under /opt/spider, including secret key, databases, archived email messages and attachments, etc. So you can simply backup whole /opt/spider/ directory with tool like "rsync".

Migration is similar, restore whole /opt/spider/ with correct owner and group (spider:spider) and permission (0700). WARNING: /opt/spider/secret.key better be 0400.

By the way, are you going to backup spider to another server via rsync? or backup to other storage like Amazon S3? Please share your use case so that we can add backup related operations in Spider web console directly (e.g. manage the backup methods / destinations on web console).

FYI https://spiderd.io/docs/files-directories.html

3 Reply by fsantiago06111979

  • fsantiago06111979
  • Member
  • Offline
  • Registered: 2014-05-04
  • Posts: 229

Re: backup / migration of existing spider archiver

this is a migration to a new server. i plan to use rsync or scp to copy files over. i also make a 2nd backup of all files to s3 storage (Wasabi).

ok, your instructions don't include the systemd (or other) service, user creation (spider), etc... so it seems to me what i need to do is:

1.> copy over my backup files from the original server
2.> install spider according to your base docs on the new server, to treat it as an update and reestablish whatever's missing (the above)
3.> anything else?

4 Reply by fsantiago06111979

  • fsantiago06111979
  • Member
  • Offline
  • Registered: 2014-05-04
  • Posts: 229

Re: backup / migration of existing spider archiver

or as a variation to my plan, reverse steps 1 & 2, followed by maybe repeating the above step 2 again afterwards (maybe). not sure what would need to be (re)established. or, i could just omit the repeat of step 2. let me know what you think please and thank you.

5 Reply by fsantiago06111979

  • fsantiago06111979
  • Member
  • Offline
  • Registered: 2014-05-04
  • Posts: 229

Re: backup / migration of existing spider archiver

i re-read the message displayed with you run the install bash and it makes me think just do:

1.> copy backup
2.> run 'bash install.sh' from the current version package v2.3.0, and it will take care of what's missing.

done. i'm working on it now...and will report back.

6 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 30,793

Re: backup / migration of existing spider archiver

fsantiago06111979 wrote:

i re-read the message displayed with you run the install bash and it makes me think just do:

1.> copy backup
2.> run 'bash install.sh' from the current version package v2.3.0, and it will take care of what's missing.

done. i'm working on it now...and will report back.

Yes.
But better this way:

- install Spider on new server.
- stop spider service.
- remove all files under /opt/spider/, including "/opt/spider/secret.key". We need old secret key.
- Run "rsync" to sync /opt/spider/ from old server to new server. Make sure you run rsync with argument "--owner --group --perms" to preserve owner/group/permission. Since Spider install script creates user/group with fixed uid/gid, rsync should set correct owner/group for you.

7 Reply by fsantiago06111979

  • fsantiago06111979
  • Member
  • Offline
  • Registered: 2014-05-04
  • Posts: 229

Re: backup / migration of existing spider archiver

i also added --executability, and will try it and report back.

8 Reply by fsantiago06111979

  • fsantiago06111979
  • Member
  • Offline
  • Registered: 2014-05-04
  • Posts: 229

Re: backup / migration of existing spider archiver

this is the final command line i settled on:

root@spider:/opt/spider# rsync -ogprE -e 'ssh -p [port number]' --progress root@[ip address]:/opt/spider/ .

i added:

-E (to preserve any possible file executability)
-r (directory recursion)
-e (necessary when using a custom ssh port)
--progress (to know what's going on and see its progress)

i gotta say, rsync was SOO much faster than scp. i had initially tried scp last night but re-did it now w/ rsync based on your recommendation. it was orders of magnitude faster.

ok, files now in place. so i then:

> restarted spiderd
> verified my firewall allowances were in place
> updated my dns records
> setup nginx reverse proxy and obtained my letsencrypt ssl cert (did it myself w/ certbot outside of spider)
> began testing it

will report back how things are working after a bit. thanks.

9 Reply by fsantiago06111979

  • fsantiago06111979
  • Member
  • Offline
  • Registered: 2014-05-04
  • Posts: 229

Re: backup / migration of existing spider archiver

ok it appears to be working ok. one more step; after all that i also had to edit, on my mail server, the transport map file, and regenerate its hash, to update the IP address to that of the new spider archiver server. and then restart postfix. and now it's good to go. thank you sir.

10 Reply by fsantiago06111979 (edited by fsantiago06111979 2023-06-17 00:56:55)

  • fsantiago06111979
  • Member
  • Offline
  • Registered: 2014-05-04
  • Posts: 229

Re: backup / migration of existing spider archiver

question; i just realized the aws cli tool doesn't copy empty folders. so when using the command 'aws s3 sync ...', it does not send two empty folders you have in spider's directory:

/opt/spider/autocert
/opt/spider/export

i assume they're not empty if i use certain features or options but i don't so they are. i can see myself forgetting they're supposed to be there when i go to restore spider in the future.

would spider fail to work if they are missing?

would spider simply create them on start if they're missing?

if i place an empty file in those 2 directories so aws s3 sync takes them, would that cause spider to have a problem?

naturtally, if we follow your plan for migrating, those folders will be created upon rsync. but if i had to restore from scratch without an rsync copy (from s3), they would be missing.

thanks.