Reports of falling bricks and buckling columns have forced evacuations in the area around a Midtown Manhattan high-rise office building, fire officials said Tuesday.