Biến đổi biểu thức dưới dấu tích phân:
$\dfrac{x^{3}}{x+\sqrt{x^{2}+1}}=\dfrac{x^{3}(\sqrt{x^{2}+1}-x)}{(\sqrt{x^{2}+1}-x)(\sqrt{x^{2}+1}+x)}$
$=\dfrac{x^{3}\sqrt{x+1}-x^{4}}{1}=x^{3}\sqrt{x^{2}+1}-x^{4}$
$I=\int\limits_{0}^{1}\dfrac{x^{3}}{x+\sqrt{x^{2}+1}}dx=\int\limits_{0}^{1}(x^{3}\sqrt{x^{2}+1}-x^{4})dx$
$=\int\limits_{0}^{1}x^{3}\sqrt{x^{2}+1}dx-\int\limits_{0}^{1}x^{4}dx=I_{1}-I_{2}$
Tính $I_{1}$: Đặt $t=\sqrt{x^{2}+1}\Rightarrow t^{2}=x^{2}+1\Rightarrow x^{2}=t^{2}-1$ và $tdt=xdx$
$x=0\Rightarrow t=1$
$x=1\Rightarrow t=\sqrt{2}$
$I_{1}=\int\limits_{0}^{1}x^{2}\sqrt{x^{2}+1}xdx=\int\limits_{1}^{\sqrt{2}}(t^{2}-1)t.tdt=\int\limits_{1}^{\sqrt{2}}(t^{4}-t^{2})dt$
$=\left. \left(\dfrac{t^{5}}{5}-\dfrac{t^{3}}{3}\right)\right|_{1}^{\sqrt{2}}=\dfrac{2\sqrt{2}-2}{15}$
Tính $I_{2}=\int\limits_{0}^{1}x^{4}dx=\left. \dfrac{x^{5}}{5}\right|_{0}^{1}=\dfrac{1}{5}$
Vậy $I=I_{1}-I_{2}=\dfrac{2\sqrt{2}-2}{15}-\dfrac{1}{5}=\dfrac{2\sqrt{2}-5}{15}$