The Sensex ended down 97 points at 15,873. Earlier in the day the index moved within a range of 16,049 - 15,799.
The Nifty ended down 29 points at 4,750.
Meanwhile in Asia, the Nikkei fell below its 25-day average and ended 0.5% lower at 8,440.
With many overseas investors on year-end holidays, market participants said it had been difficult to buy risk assets though the re-opening of stock trading in the United States later in the day could provide more direction.
Markets are likely to remain volatile for the next few days as traders roll over positions in the derivatives segment on Thursday.
Foreign funds bought shares worth Rs 113.43 crore on Monday as per provisional data from the stock exchanges.
"Today we tested the psycholical mark of 4800 (high 4801) and then have been in sideways territory.
Holiday season and testing of a pyschological as well as technical target has led to lowering of momentum.
However as its the last week of the year and also Derivatives expiry, momentum is expected to return for the rest of the week," said Ranak Merchant, Technical Analyst - Strategies, Sushil Financial Services.
All the sectoral indices were in red. Realty shares fell on profit booking after recent gains.
BSE realty index dropped 1.6% to 1,427.
Metal, power and bankex also dropped over 1% each in trades.
Reliance was the top dragger along with Tata Motors and Infosys - accounting for a 34
Markets end higher led by IT, Infosys up 3%
5 corporate honchos who will rule Indian business in 2012
CNG may be hiked by Rs 2/kg
Hurry! Book your car and get huge DISCOUNTS
Markets start Dec on a promising note