Extreme waves may destroy the coast‐protection structures through continuous scour around them. A 3D incompressible Smoothed Particle Hydrodynamics (ISPH) erosion model is proposed to simulate the scouring process around coastal structures. The erosion model is based on the turbidity water particle concept and the sediment motion is initiated when the fluid bottom shear stress exceeds the critical value. To validate the developed model, a laboratory flume experiment was carried out to study the clear water scouring around a vertical cylinder under unidirectional current, in which high‐speed video cameras were used for the real‐time monitoring of sediment movement. Recently, the Incompressible Smoothed Particle Hydrodynamics (ISPH) method solving the 2D RANS (Reynolds Averaged Navier‐Stokes) equations with the k–ε turbulence closure is constructed. The concept of “massless ISPH” utilizing the definition of "particle density" (number of computational particles within unit volume) is stressed. The skills of this numerical model are tested by applying to two laboratory experiments: (1) A non‐breaking solitary wave propagating over a bottom‐mounted barrier and (2) a solitary wave breaking on a 1 on 50 slope