Yuan, Shao and Liu proved that the H-shape tree $H'_n =P_{1,2;n-3}^{1,n-6}$ minimizes the spectral radius among all graphs with order $n\geqslant 9$ and diameter $n-4$. In this paper, we achieve the spectral characterization of all graphs in the set $\mathscr{H}' = \{ H'_n\}_{n\geqslant 8}$. More precisely we show that $H'_n$ is determined by its spectrum if and only if $n \neq 8, 9,12$, and detect all cospectral mates of $H'_8$, $H'_9$ and $H'_{12}$. Divisibility between characteristic polynomials of graphs turns out to be an important tool to reach our goals.