In IMAP's RFC (6.3.1 SELECT Command) is described a one element of SELECT command response:
OK [UNSEEN <n>] The message sequence number of the first unseen message in the mailbox. If this is missing, the client can not make any assumptions about the first unseen message in the mailbox, and needs to issue a SEARCH command if it wants to find it.
So, I'm just thinking loud. Can we assume that there are no unseen messages in the mailbox when server doesn't include UNSEEN in the response? This is true for dovecot, how about others? We could skip one SEARCH call.