So I partially succeedded. A minimal requirement for LDAP/QNAP integration is objectCalss=posixAccout, uidNumber and gidNumber attributes for user entry. As I metioned, you can specify all in /etc/config/nss_ldap.conf (and additional filters also), but UI wizard will not respect your changes.
=nss_ldap.conf ===================
host ldap.mydomain.org
base dc=mydomain,dc=org
uri ldap://mail.mydomain.org/
ssl off
rootbinddn cn=vmailadmin,dc=mydomain,dc=org
nss_base_passwd ou=Users,domainName=mydomain.org,o=domains,dc=mydomain,dc=org?one?(accountStatus=active)
nss_base_shadow ou=Users,domainName=mydomain.org,o=domains,dc=mydomain,dc=org?one
nss_base_group ou=Groups,domainName=mydomain.org,o=domains,dc=mydomain,dc=org?one
tls_checkpeer no
bind_policy soft
bind_timelimit 2
tls_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:!MD5
nss_initgroups_ignoreusers admin
================================
you should also add user filter to /etc/afp.conf
===============================
user filter (accountStatus=active)
===============================
With this scenario you can make AFP, NFS and Web UI work with your LDAP, but not Samba. QNAP Samba fully relies on AD objectClasses like sambaAccount and sambaDomain, so you need fully functional AD controller (maybe samba4), but not LDAP.
In short, QNAP Samba does not support LDAP, its AD only.
Here is OpenLDAP logs
==================================================
ldap slapd[6782]: conn=1001 op=1 SEARCH RESULT tag=101 err=0 qtime=0.000024 etime=0.000167 nentries=1 text=
ldap slapd[6782]: conn=1001 op=2 SRCH base="dc=npl,dc=ru" scope=2 deref=0 filter="(&(?objectClass=sambaDomain)(?sambaDomainName=NPL))"
ldap slapd[6782]: conn=1001 op=2 SRCH attr=sambaDomainName sambaNextRid sambaNextUserRid sambaNextGroupRid sambaSID sambaAlgorithmicRidBase objectClass
ldap slapd[6782]: conn=1001 op=2 SEARCH RESULT tag=101 err=0 qtime=0.000023 etime=0.000667 nentries=0 text=
ldap slapd[6782]: conn=1001 op=3 SRCH base="dc=npl,dc=ru" scope=2 deref=0 filter="(&(?sambaDomainName=NPL)(?objectClass=sambaDomain))"
ldap slapd[6782]: conn=1001 op=3 SRCH attr=sambaDomainName sambaNextRid sambaNextUserRid sambaNextGroupRid sambaSID sambaAlgorithmicRidBase objectClass
ldap slapd[6782]: conn=1001 op=3 SEARCH RESULT tag=101 err=0 qtime=0.000025 etime=0.000156 nentries=0 text=
ldap slapd[6782]: conn=1001 op=4 do_add: invalid dn (sambaDomainName=NPL,dc=npl,dc=ru)
ldap slapd[6782]: conn=1001 op=4 RESULT tag=105 err=34 qtime=0.000022 etime=0.000295 text=invalid DN
ldap slapd[6782]: conn=1001 fd=12 closed (connection lost)
==========================
And here is corresponding QNAP Samba daemon log
==========================
../../source3/passdb/pdb_ldap.c:6753(pdb_ldapsam_init_common)
pdb_init_ldapsam: WARNING: Could not get domain info, nor add one to the domain. We cannot work reliably without it.
../../source3/passdb/pdb_interface.c:184(make_pdb_method_name)
pdb backend ldapsam:ldap://mail.npl.ru did not correctly init (error was NT_STATUS_CANT_ACCESS_DOMAIN_INFO)
../../source3/smbd/server.c:2008(main)
smbd version 4.13.17 started.
Copyright Andrew Tridgell and the Samba Team 1992-2020
========================