There already exists some quota support for samba both on client side and server side.
- On the client side there is the smbcquota tool, afaics this only works using SMB1 (or at least I only ever saw it working with smb1, I don't see any cmdline switches to change it)
- On the server side there is support for scripts and/or ability to hook into standard linux system quota api(s) if they exist
From the windows client you can additionally manage the quotas (with the appropriate user e.g. Administrator privileged user on windows or root to manage a linux share).
This all seems to work fine as long as you are using SMB1, using SMB2 and it falls down, it seems there are some missing pieces, at least a SMB2_QUERY_INFO message with class SMB2_GETINFO_QUOTA (and I guess the equivalent SETINFO) don't seem to be implemented. Also SMB2_QUERY_INFO message with class FS_INFO & level FileFsControlInformation fails for the special Quota stream '\$Extend\$Quota:$Q:$INDEX_ALLOCATION'
Additionally wireshark doesn't dissect the SMB2_QUERY_INFO message with class SMB2_GETINFO_QUOTA request (or associated response) so it isn't even clear what information being passed back and forth with normal windows traffic
- firstly improve the existing SMB2 wireshark dissector to try and parse the missing quota related information
- use the info above to try and get SMB2 quota related communication running (to allow remote get/set/list of quota limits from windows client against a linux samba share)
- try and port (or just add support for SMB2) to smbcquota tool
This project is one of its kind!