1 Topic by angeloklin

  • angeloklin
  • Member
  • Offline
  • From: Australia
  • Registered: 2019-10-03
  • Posts: 52

Topic: Reduce LLVM build time

==== REQUIRED BASIC INFO OF YOUR IREDMAIL SERVER ====
- iRedMail version (check /etc/iredmail-release): Unreleased 1.3.2 from Github
- Deployed with iRedMail Easy or the downloadable installer? Download from Github
- Linux/BSD distribution name and version: FreeBSD 12.1-RELEASE-p10
- Store mail accounts in which backend (LDAP/MySQL/PGSQL): PGSQL 12.x
- Web server (Apache or Nginx): Nginx
- Manage mail accounts with iRedAdmin-Pro? No
- [IMPORTANT] Related original log or error message is required if you're experiencing an issue.
====

On FreeBSD all packages are build from the `Ports`.
The LLVM is build as a dependency for PostgreSQL.
As there is no explicit configuration, all the defaults are used.
The defaults are likely to be configured as to be used for development, opposed to as a dependency.
For development there are cross-ompilers, debuggers, (extensive) documentation.
The build of LLVM takes a good chunk of the iRedMail installation (six hours in one test)

I checked and asked in the FreeBSD foruns and got some feedback. Link below:
Bare minimum build of LLVM

Below is a suggestions (TO BE Tested!)

# cd /usr/ports/devel/llvm90
# make config
┌────────────────── llvm90-9.0.1_2 ───────────────────┐
│ ┌────────────────────────────────────────────┐ │  
│ │+[x] BE_AMDGPU    AMD GPU backend (required by mesa)                      │ │  
│ │+[x] CLANG        Build clang                                             │ │  
│ │+[ ] COMPILER_RT  Sanitizer libraries                                     │ │  
│ │+[ ] DOCS         Build and/or install documentation                      │ │  
│ │+[ ] ELFV2        [PowerPC64] Use OpenPOWER ELF ABI v2 by default         │ │  
│ │+[ ] EXTRAS       Extra clang tools                                       │ │  
│ │+[ ] GOLD         Build the LLVM Gold plugin for LTO                      │ │  
│ │+[x] LIT          Install lit and FileCheck test tools                    │ │  
│ │+[x] LLD          Install lld, the LLVM linker                            │ │  
│ │+[ ] LLDB         Install lldb, the LLVM debugger                         │ │  
│ │+[x] LLD_LINK     Link ld.lld as ld to clang uses it                      │ │  
│ │+[x] OPENMP       Install libomp, the LLVM OpenMP runtime library         │ │  
│ │+[ ] PYCLANG      Install python bindings to libclang                     │ │  
│ │──────────────────── BACKENDS ──────────────────│ │  
│ │+( ) BE_FREEBSD   Backends for FreeBSD architectures                      │ │  
│ │+(*) BE_NATIVE    Backend(s) for this architecture (X86)                  │ │  
│ │+( ) BE_STANDARD  All non-experimental backends                           │ │  
│ └────────────────────────────────────────────┘ │  
└───────────────────────────────────────────────┘

----

Spider Email Archiver: On-Premises, lightweight email archiving software developed by iRedMail team. Supports Amazon S3 compatible storage and custom branding.

2 Reply by angeloklin (edited by angeloklin 2020-10-19 16:56:05)

  • angeloklin
  • Member
  • Offline
  • From: Australia
  • Registered: 2019-10-03
  • Posts: 52

Re: Reduce LLVM build time

Tested the reduced configuration of LLVM.

The original build time of LLVM alone, with default, all included setup was 6:08:08. With the reduced configuration the time was 2:49:39.

Testing environment:
- macOS Mojave 10.14.6
- VMware Fusion 10.1.6 (VM 4GB, 20GB)
- iRedMail 1.3.2 (from Github)
- FreeBSD 12.1, Postgres 12.x

NOTE: I am testing iRedMail installation focus on the building of the packages and overall install process. I am not testing functionality either manually or with any regression test or script. If iRedMail's team likes the idea, I would assume they have proper testing sets.

The modifications were:

# = = = = = = = = = =
# ./functions/package_freebsd.sh
. . .
freebsd_make_conf_plus_option 'LICENSES_ACCEPTED' 'DCC'

    for p in \
        . . .
        devel_cmake \
        devel_llvm90 \           ### add this line
        . . .
        www_sogo4; do
        mkdir -p /var/db/ports/${p} >> ${INSTALL_LOG} 2>&1
    done

    # cmake. DEPENDENCE
    cat > /var/db/ports/devel_cmake/options <<EOF
OPTIONS_FILE_UNSET+=DOCS
OPTIONS_FILE_UNSET+=MANPAGES
EOF

    # llvm DEPENDENCE            ### add this block
    cat > /var/db/ports/devel_llvm90/options <<EOF
OPTIONS_FILE_UNSET+=BE_AMDGPU
OPTIONS_FILE_SET+=CLANG
OPTIONS_FILE_UNSET+=COMPILER_RT
OPTIONS_FILE_UNSET+=DOCS
OPTIONS_FILE_UNSET+=ELFV2
OPTIONS_FILE_UNSET+=EXTRAS
OPTIONS_FILE_UNSET+=GOLD
OPTIONS_FILE_SET+=LIT
OPTIONS_FILE_SET+=LLD
OPTIONS_FILE_UNSET+=LLDB
OPTIONS_FILE_SET+=LLD_LINK
OPTIONS_FILE_SET+=OPENMP
OPTIONS_FILE_UNSET+=PYCLANG
OPTIONS_FILE_UNSET+=BE_FREEBSD
OPTIONS_FILE_UNSET+=BE_STANDARD
OPTIONS_FILE_SET+=BE_NATIVE
EOF
. . .

3 Reply by angeloklin

  • angeloklin
  • Member
  • Offline
  • From: Australia
  • Registered: 2019-10-03
  • Posts: 52

Re: Reduce LLVM build time

This option is so simple that it hurts, not considering it in the first place. I guess we do not want to rock the boat by removing things.

Simply disable LLVM for Postgres. Thanks for the pople in the FreeBSD forum for the idea.
Bare minimum build of LLVM

Postgres uses LLVM since version 11 to optimise queries by creating specific functions on the fly.
Some reference from the documentation: 31.1. What Is JIT compilation?

It is a choice (as always). Some might prefer to have this ON and benefit from this feature. Some might prefer to turn it OFF. Criteria can be the hardware configuration available. Compilation time vs running time. Size and flow of messages on the mail server.
I run a $4 VCS with 1GB memory on a personal domain. I really do not need to wait six hours for LLVM to compile, and I do not believe I would benefit from this. Someone else's needs would be different.

The specifics:
- Build time for LLVM: 0:00:00. Postgres is one think that link LLVM as a dependency. Turn that OFF, and you do not need LLVM at all.

# = = = = = = = = = =
# ./functions/packages_freebsd.sh
. . .
    # PostgreSQL
    cat > /var/db/ports/databases_postgresql${PGSQL_VERSION}-server/options <<EOF
. . .
OPTIONS_FILE_SET+=LLVM      # Change FROM SET
OPTIONS_FILE_UNSET+=LLVM    # TO UNSET

ClamAV also offers to build with LLVM, but it is UNSET by default

# = = = = = = = = = =
# ./functions/packages_freebsd.sh
    # ClamAV. REQUIRED.
    cat > /var/db/ports/security_clamav/options <<EOF
. . .
OPTIONS_FILE_SET+=DOCS      # BTW: This should be UNSET
. . .
OPTIONS_FILE_UNSET+=LLVM    # Default is already UNSET

Good luck and stay safe.

4 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Reduce LLVM build time

Good catch, I think we can disable llvm directly. Will commit this change later.

5 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Reduce LLVM build time

After removed LLVM, the installation toke about 70 minutes for PGSQL backend on my local testing machine. smile

6 Reply by angeloklin

  • angeloklin
  • Member
  • Offline
  • From: Australia
  • Registered: 2019-10-03
  • Posts: 52

Re: Reduce LLVM build time

ZhangHuangbin wrote:

After removed LLVM, the installation toke about 70 minutes for PGSQL backend on my local testing machine. smile

Sounds great.

In case you do not follow too closelly, FreeBSD 12.2 is scheduled to be released in a week's time, on the 27/Oct.

FreeBSD 12.2 Release Process

Looking forward to upgrade (reinstall) my systems with FreeBSD 12.2 plus iRedMail 1.3.2, and leave Python 2.7 behind for good.

BTW: Is the conversion from Py2.7 to Py3.8 close to be finished and v1.3.2 soon after?

Also, I have sent some other suggestions on configs for the `options`.

Cheers
Keep it safe.

7 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Reduce LLVM build time

angeloklin wrote:

In case you do not follow too closelly, FreeBSD 12.2 is scheduled to be released in a week's time, on the 27/Oct.

I planned to release iRedMail-1.3.2 (or 1.4.0) this week.
OpenBSD 6.8 was out days ago, and new iRedMail release supports it too.
I will try to setup FreeBSD 12.2 locally and catch up, this way new iRedMail might be delayed for 1 week (installing from ports takes too long).

angeloklin wrote:

BTW: Is the conversion from Py2.7 to Py3.8 close to be finished and v1.3.2 soon after?

Py 2to3 migration has been finished, including iRedAPD, mlmmjadmin, iRedAdmin(-Pro), they will be shipped in upcoming iRedMail release.

angeloklin wrote:

Also, I have sent some other suggestions on configs for the `options`.

Please share them here, we still have some time.

8 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Reduce LLVM build time

Another plan is we tag new iRedMail release before FreeBSD 12.2 announced, and re-tag it with commits to support 12.2.

9 Reply by pacohope

  • pacohope
  • Member
  • Offline
  • Registered: 2017-07-24
  • Posts: 17

Re: Reduce LLVM build time

Joining this thread a little late, but angeloklin you can save a lot of build time by installing packages. You can do pkg install llvm and skip that build. There are a LOT of packages that can be installed from binary. But, frustratingly, not all.

The install script is really good about picking up where it stopped. It also handles the situation where something is already installed.

When I build, I often do a lot of pkg install of basics (e.g., python, Imagemagick, protobuf, and other low-level libraries). Then I run the install script. When I see it start to build something basic from source (e.g., apache or nginx), I Ctrl-C the install script, do pkg install apache24, and then run install again. I've made mistakes doing this, though, so I can't give you a complete list.

Most of the important mail-related binaries must be built from source (postfix, dovecot, etc.). You can install apache or nginx as a binary, uwsgi, postgresql, mariadb, or openldap and lots of other things, but not everything. I am working on upgrading my own system soon, and I want to make a list of which packages are probably installable from binary.

I like your idea of changing the build options for llvm, but it would be simpler to just do 'pkg install postgresql' and never deal with LLVM in the first place. It would be tons faster and get you the same result.

10 Reply by angeloklin

  • angeloklin
  • Member
  • Offline
  • From: Australia
  • Registered: 2019-10-03
  • Posts: 52

Re: Reduce LLVM build time

pacohope wrote:

Joining this thread a little late, but angeloklin you can save a lot of build time by installing packages. You can do pkg install llvm and skip that build. There are a LOT of packages that can be installed from binary. But, frustratingly, not all.

The install script is really good about picking up where it stopped. It also handles the situation where something is already installed.

When I build, I often do a lot of pkg install of basics (e.g., python, Imagemagick, protobuf, and other low-level libraries). Then I run the install script. When I see it start to build something basic from source (e.g., apache or nginx), I Ctrl-C the install script, do pkg install apache24, and then run install again. I've made mistakes doing this, though, so I can't give you a complete list.

Most of the important mail-related binaries must be built from source (postfix, dovecot, etc.). You can install apache or nginx as a binary, uwsgi, postgresql, mariadb, or openldap and lots of other things, but not everything. I am working on upgrading my own system soon, and I want to make a list of which packages are probably installable from binary.

I like your idea of changing the build options for llvm, but it would be simpler to just do 'pkg install postgresql' and never deal with LLVM in the first place. It would be tons faster and get you the same result.

Thanks for the reminder. I am aware that one could take that approach, but best practices and more experienced advice on FreeBSD is to avoid mixing pkg and ports. I install FreeBSD from scratch with the very minimum and use pkg to install Bash, which is required by iRedMail.

A secondary, but still strong reason is for maintenance/updates. Using both pkg and ports forces the updates to be made individually, which in the ends takes longer. I have some personal domains, so I do not need to spend much admin time on the servers, but I do an update every now and then, maybe once a quarter on average. I use one line portmaster to do a full upgrade.

I am considering studying and move to use poudriere as you can build a bundle and use that on other servers, which is handy if you have more than one, which I do.

Cheers

11 Reply by ZhangHuangbin

  • ZhangHuangbin
  • ZhangHuangbin
  • iRedMail Developers
  • Offline
  • Registered: 2009-05-06
  • Posts: 31,088

Re: Reduce LLVM build time

I realized that we can install the relied packages with pkg at the beginning, but as @angeloklin mentioned, not a good idea to mix pkg + ports, so i have to go the full ports way.

Another solution is, we (iRedMail team) build binary packages on a dedicated server and offer them publicly for all iRedMail + FreeBSD users. but is it a good idea?

12 Reply by angeloklin

  • angeloklin
  • Member
  • Offline
  • From: Australia
  • Registered: 2019-10-03
  • Posts: 52

Re: Reduce LLVM build time

ZhangHuangbin wrote:

I realized that we can install the relied packages with pkg at the beginning, but as @angeloklin mentioned, not a good idea to mix pkg + ports, so i have to go the full ports way.

Another solution is, we (iRedMail team) build binary packages on a dedicated server and offer them publicly for all iRedMail + FreeBSD users. but is it a good idea?

Hello,

  For the users of iRedMail this would be great, and it could work by having a server with the binaries using some solution as Poudriere (BTW: the same FreeBSD uses itself). It would be something akin to the docker version, by means of serving pre-cooked binaries with the specific necessary configuration.
  The BUTs are:
  - Need to compile, perhaps once a month or every time there is a new version or fix.
  - Need to provide a server to serve the binaries.
  - Need to revamp the existing installation code or clone the one from other Linux distributions and adapt to the new format.
  - A potential catch is that updates would have to be done from the same server.
  - Need to keep the packages as update as they are in the ports.

  It is completely viable and feasible but changes the paradigm, which in the end is a decision for the iRedMail team to make.

  Personally, I don't mind anyway as long as the overall solution works. I am comfortable enough to solve some problems myself (e.g. I have enabled fail2ban). It is possible that most FreeBSD users are also capable to the same extent, but it is not guaranteed that they are or they want to take that responsibility.

  I would stress again the best practice of not mixing and matching pkg and port. As per the community's wisdom on this matter: One or the other, avoid both. One can get by during installation, but it comes back to bite you when you need to upgrade packages.

Cheers, and stay safe.