"Even on this very day - today - 188 new regulations come into force. While other countries make products; we sometime just make rules - rules that cost British business £56bn a year. We will cut that cost," he told the Conservative party conference in Blackpool.
The Conservatives are believed to be preparing for a November election, although the Government has yet to announce a date for the next election, which could be at any time up to 3 June 2010.
Duncan says he will change the name of the department, and use it to eke out unwanted regulation.
"We will reduce regulation by defining a coherent concept of what regulation actually means, and we will inject sense into our understanding of risk.
"We will make this Department the clearing house across Whitehall for all legislation which has a regulatory impact," he said.
"We will ensure that for any regulation that is introduced two or more existing ones will be removed."