The final Markit/Nikkei services Purchasing Managers' Index (PMI) for Japan increased to 53.7 in August from 51.2 in July.
A reading below 50 indicates contraction of activity.
The index was driven by a rise new business, while cost pressures eased.
"Activity growth accelerated to the strongest since October 2013, underpinned by a solid increase in new orders. In contrast, employment levels declined, offsetting the slight increase seen at the start of Q3. That said, the rate of job shedding was marginal," economist at Markit, Amy Brownbill, said.