The irreducible decomposition of scalar holomorphic discrete series representations when restricted to semisimple symmetric pairs (G, H) is explicitly known by Schmid [Die Randwerte holomorphe funktionen auf hermetisch symmetrischen Raumen, Invent. Math.9 (1969–1970) 61–80] for H compact and by Kobayashi [Multiplicity-Free Theorems of the Restrictions of Unitary Highest Weight Modules with Respect to Reductive Symmetric Pairs, Progress in Mathematics, Vol. 255 (Birhäuser, 2007), pp. 45–109] for H non-compact. In this paper, we deal with the symmetric pair (U(n, n), SO* (2n)), and extend the Kobayashi–Schmid formula to certain non-tempered unitary representations which are realized in Dolbeault cohomology groups over open Grassmannian manifolds with indefinite metric. The resulting branching rule is multiplicity-free and discretely decomposable, which fits in the framework of the general theory of discrete decomposable restrictions by Kobayashi [Discrete decomposability of the restriction of A𝔮(λ) with respect to reductive subgroups II — micro-local analysis and asymptotic K-support, Ann. Math.147 (1998), 709–729].
Commuting Toeplitz operators on bounded symmetric domains and multiplicity-free restrictions of holomorphic discrete series
