The rest mass of a particle is a term used in relativity to denote the mass that a particle has at rest. More specifically, it is the mass an observer would measure a particle to have if the observer is in the same inertial frame of reference as the particle in question. The term is used to draw a distinction with the other term "relativistic mass". Rest mass is also sometimes called "invariant mass".
The rest mass of a particle (the terms are almost always used in the context of atoms or subatomic particles) is what it would weigh if you put it on a scale. It includes any contribution from the particle's potential energy. So, for example, the mass of a Radium-226 atom is 226.0254098 amu. This includes the .0052288 amu for the potential energy that would be released if the atom undergoes an alpha decay, as well as the 226.020181 amu that are latent in the results of the alpha decay. If you could put a radium atom on a scale, it would register 226.0254098 amu.
It happens that, under relativity, the momentum of a particle in motion is given by
instead of the "classical" formula
where v is the particle's speed. This definition is required in order to get precise conservation of momentum in all frames of reference.
As can be seen, the difference is only significant for particles travelling at speeds comparable to the speed of light, that is, "relativistic" speeds.
Some people have gotten around this issue by using a subscript zero to denote the particle's rest mass, and defining something called the "relativistic mass", denoted m, as
so that the momentum formula will look more natural in terms of this "relativistic mass" m.
This usage is obsolete and should be avoided. It's best to think of momentum as something that grows faster than just 
would suggest.
The factor
shows up in a number of places in relativity, and is called the Lorentz factor, commonly denoted with the Greek letter 
. This means that the formula for momentum is
