If it was an asphalt track you would essentially be off track. The games idea is prepared surface versus outside surface. The punishment is supposed to be the same as grass even though generally on a track flat inside would still be paved mostly.
Rather than put a wall there for you to hit...