We report on the direct measurement of the elastic scattering time $\tau_S$ of ultracold atoms propagating in optical disordered potentials. By exploring this fundamental quantity over a large range of experimental parameters, we observe variations of $\tau_S$ over more than three orders of magnitude, in excellent agreement with numerical calculations. It allows us to study the crossover from the weak to the strong scattering regimes, which are explicitly identified by a comparison to the first order Born approximation. We especially discuss the relevance of the widely used criterion $k l_S \sim 1$ to locate this crossover. While it is validated for a Gaussian disorder, it breaks down for the laser speckle disorders used in the experiments, where large deviations to Born predictions are observed. This result highlights the strong influence of the disorder statistics on the crossover and, more generally, on the behavior of the time $\tau_S$ in the strong scattering regime.