\[2ab+(2a-7b)^2=2ab+4a^2-28ab+49b^2=4a^2-26ab+49b^2.\]
При \(a=\sqrt5\), \(b=\sqrt8=2\sqrt2\):
\[4\cdot5-26\sqrt{40}+49\cdot8=20-52\sqrt{10}+392=412-52\sqrt{10}.\]
Ответ: \(412-52\sqrt{10}\).