Back in 2010 Google chief executive Eric Schmidt noted that the amount of data collected since the dawn of humanity until 2003 was the equivalent to the volume we now produce every two days.

This data comes not only from posts to social media sites, mobile signals and purchase transactions but increasingly from sensors on objects from lamp-posts to skyscrapers.

The so-called internet of things offers a new way to analyse and measure city life, from whether water pipes are leaking to how traffic is flowing on the roads and whether buildings are using energy in the most efficient way.

A prediction from architect and urban planner Prof Mike Batty back in 1997 that by 2050 everything around us would be some form of computer is already starting to come true.

In Birmingham, lamp-posts are being fitted with sensors that can transmit information about cloud cover to offer hyper-local weather forecasting.

In Norway, more than 40,000 bus stops are tweeting, allowing passengers to leave messages about their experiences, and in London the mayor’s office has just begun a project to tag trees so that people can learn about their history.

Sensors on objects also allow people to tell stories about city life in whole new ways.

At MIT’s Senseable City Lab, 5,000 pieces of rubbish in Seattle were geo-tagged and tracked around the country for three months to find out whether recycling was really efficient.

The project proved three things, Carlo Ratti, head of the lab, told the BBC.

“The project showed us that we need to design a better system because some of this stuff was moving thousands of miles in the wrong direction.

“It also promoted behavioural change. We shared the information with citizens and one guy stopped drinking water in plastic bottles when he saw that [they were] all going straight to landfill.”

A third, more unexpected result of the trial happened when a burglar stole some of the tags from the labs.

“It was very funny and it didn’t take the police long to trace him,” said Dr Ratti.

Now the team is working on a project in Singapore which offers advanced weather forecasts to taxi drivers to get them to places in the city ten minutes before it starts to rain.

Big data is not always such fun.

Supermarket chain Tesco is installing sensors across its stores to reduce heating and lighting costs.

The records of the fridge systems in one store alone produce 70 million data points a year.

“How do you make sense of that amount of data?” asked John Walsh, energy manager for Tesco Ireland.

And in this case the only objective is to run the fridges more efficiently.

Escalate that to a city, with data from transport systems, utilities, rubbish collection, hospitals, schools, offices and government, and the scale of the problem is obvious.

Tesco turned to IBM’s data analytics, also used by many cities.

Computer-aided design company Autodesk has been working with San Francisco, Vancouver and Bamberg, in southern Germany, to build 3D visualisations over which government can overlay data sets to see how a city is performing at any time.

Presenting data in new ways has had surprising consequences.

In Germany the model was used to show people what the impact of a new railway line would be.

“It was almost like a gaming environment. People could put it on their iPad and zoom around the model,” said Phil Bernstein, vice-president of industry relations at Autodesk.

Previously anyone wanting to get involved in city planning would have had to make a trip to the town hall to look at maps that they may not have understood. Making them have much more visual impact could revolutionise how cities are built, he thinks.

“It is a political act as well as a technical act. It makes the decision-making process transparent and democratic, and the design process more inclusive.”

Whether we like it or not, we are already starting to interact with our cities, whether via the text message that offers you 20% off purchases from the shop you have just walked past or location-sensing apps on our smartphones that tell us about the nearest coffee bar.

“My phone knows that I normally work until 17:30 and knows the next bus I should catch before I even ask it. It is beginning to predict my life,” said Andrew Hudson-Smith, who heads University College of London’s Centre for Advanced Spatial Analysis.

“I could turn it off but I don’t because I find it very useful but I am basically giving everything to Google. It is fantastic but scary at the same time and shifts the way the world works,” he said.

His lab has been at the forefront of some big data collection projects, and while he sees huge value in it for researchers he recognises that it raises huge ethical issues.

“We can collect every tweet. People have clicked “yes” to those terms but don’t realise that everything you share can be collected. We could be walking blindly into a 24/7 surveillance society,” he said.

Nick Pickles, director of privacy campaign group Big Brother Watch, agrees that big data could become just another word for Big Brother.

“The core functionality of a smart city requires a vast amount data to be collected on every aspect of our lives every minute of every day. The question is how does that data get used? And it doesn’t require a huge amount of imagination to see how it could be used to monitor people in a very real Big Brother way.

“At present the control of information is being taken away from citizens, and companies providing services are rushing to find ways of generating revenue from the data they hold. The danger is that when smart cities become a reality, individuals will not be able to control the ways they are monitored or what happens to the information, which is exactly the opposite of how it should be.”

The issue has huge implications for society and is going to need serious debate, Dr Ratti believes.

“We do need to think about how we want tomorrow’s society to work but it is a bigger discussion than just smart cities,” he said.

“We are basically building a digital copy of our physical world and that is having profound consequences.”