Footage shows the building crumbling to the ground after it exploded in a huge ball of fire and smoke.
More than 150 firefighters are battling the blaze.
Local media outlets reported that the building was a headquarters for the federal police but had been abandoned for many years and was frequently occupied by squatters.
According to local reports, one man died while being rescued moments before the building came down.
CBS News reported that the building's collapse rained fiery debris onto nearby structures and the street below as firefighters worked to evacuate people from the area and create a safe perimeter around the location.
Romulo de Souza, 49, who lived in a neighboring building, said that when the fire broke out, families began evacuating.
"Happily the majority got out", he said.
Capt. Marcos Palumbo of the São Paulo fire brigade said that alterations made by the residents had allowed the blaze to spread more rapidly.
Debris of a building which was engulfed by a fire and collapsed, in Sao Paulo.
"I was sleeping, I woke up my husband shouting 'fire, fire, fire.'" I took my son and left, I could not save his documents, I could save only mine", a woman named Crivalda said to a TV reporter at the scene.
Firefighters believe clearing the rubble, which was still smouldering hours afterward, could take days.
The fire and collapse are sure to put a spotlight on occupations in Sao Paulo, South America's largest city.
Dozens of homeless families were squatting in the building, according to local media.
The city and state governments have been working for years to forcibly remove squatters from buildings in central Sao Paulo, with plans for regenerating the area. "Fifteen or 20 minutes later the building fell".
- USA millennials' support for Democrats wanes with eye to midterm elections
- Can Manchester United win the Premier League this season?
- Stormy Daniels files defamation lawsuit against Trump
- Stephen Curry rejoins Golden State Warriors for Game 2
- Frank Lampard thinks Steven Gerrard will 'flourish' at Rangers
- Junket tourism
- Rockets blast off to 1-0 lead over Jazz
- San Antonio's Ada Vox eliminated from 'American Idol' competition
- Bill Cosby juror reveals Cosby's own comments led to conviction
- May Day protesters demand better rights for workers