The centre and number eight for the Stormers and Lions respectively have, according to SuperSport, told coach Neil Powell of their decision.
“Unfortunately for us, both Damian and Warren withdrew from the Olympics,” Powell told the website in a setback ahead of the Paris event.
Powell added: “It is a bit of a blow for us, but I am still confident that we have more than enough players to cover the positions that Warren and Damian played in and obviously they still had to come in and prove they could play Sevens.
“Both of them would have played in the forwards, and I am still happy with the forwards we have for those tournaments. The team we have will be strong enough to take a good team to Rio.”
South Africa currently sit in second position on the World Rugby Sevens Series standings with just two rounds remaining, in Paris and London.