AbstractWe consider the family of Hecke triangle groups $$ \Gamma _{w} = \langle S, T_w\rangle $$
Γ
w
=
⟨
S
,
T
w
⟩
generated by the Möbius transformations $$ S : z\mapsto -1/z $$
S
:
z
↦
-
1
/
z
and $$ T_{w} : z \mapsto z+w $$
T
w
:
z
↦
z
+
w
with $$ w > 2.$$
w
>
2
.
In this case, the corresponding hyperbolic quotient $$ \Gamma _{w}\backslash {\mathbb {H}}^2 $$
Γ
w
\
H
2
is an infinite-area orbifold. Moreover, the limit set of $$ \Gamma _w $$
Γ
w
is a Cantor-like fractal whose Hausdorff dimension we denote by $$ \delta (w). $$
δ
(
w
)
.
The first result of this paper asserts that the twisted Selberg zeta function $$ Z_{\Gamma _{ w}}(s, \rho ) $$
Z
Γ
w
(
s
,
ρ
)
, where $$ \rho : \Gamma _{w} \rightarrow \mathrm {U}(V) $$
ρ
:
Γ
w
→
U
(
V
)
is an arbitrary finite-dimensional unitary representation, can be realized as the Fredholm determinant of a Mayer-type transfer operator. This result has a number of applications. We study the distribution of the zeros in the half-plane $$\mathrm {Re}(s) > \frac{1}{2}$$
Re
(
s
)
>
1
2
of the Selberg zeta function of a special family of subgroups $$( \Gamma _w^N )_{N\in {\mathbb {N}}} $$
(
Γ
w
N
)
N
∈
N
of $$\Gamma _w$$
Γ
w
. These zeros correspond to the eigenvalues of the Laplacian on the associated hyperbolic surfaces $$X_w^N = \Gamma _w^N \backslash {\mathbb {H}}^2$$
X
w
N
=
Γ
w
N
\
H
2
. We show that the classical Selberg zeta function $$Z_{\Gamma _w}(s)$$
Z
Γ
w
(
s
)
can be approximated by determinants of finite matrices whose entries are explicitly given in terms of the Riemann zeta function. Moreover, we prove an asymptotic expansion for the Hausdorff dimension $$\delta (w)$$
δ
(
w
)
as $$w\rightarrow \infty $$
w
→
∞
.
The Selberg trace formula and Selberg zeta-function for cofinite Kleinian groups with finite-dimensional unitary representations
