A Special Move is any move performed by pressing the B button (or whatever button the player assigns in Brawl). Generally, Special Moves provide more damage than Smash Attacks, although it's unlikely to provide a KO in some situations.
Unlike other attacks, Special Moves can have wildly different functions: Falco's Side Special Move is a quick running swipe familiar to a Dash Attack, whereas Samus' is a projectile that leaves Samus stationary. Usually, only Special Moves can be projectiles (an exception is Snake's and Olimar's Up Smash), and some are able to render the user helpless if used in midair (usually those that function as mid-air jumps, like most Up Special Moves). A Final Smash is a special type of special move that can be accessed when the Smash Ball is obtained. When obtained, the Standard Special Move is then replaced with the Final Smash, which usually does a very high damaging attack that can affect more than one person.
In Super Smash Bros. for Nintendo 3DS / Wii U, characters special moves can be customized (up to three different customizations per move).