Novell has rejected the latest bid for its business, describing the $5.75 (£3.83) per share offer from Elliott Associates as "inadequate".
The firm said that, after careful consideration, it has decided that the cash offer "undervalues the company's franchise and growth prospects".
Novell will now look at the alternatives as it struggles to get more money for its business.
The company said in a statement that its board of directors had approved plans to review "various alternatives" to this offer to "enhance stockholder value".
Novell added that these alternatives could include strategic partnerships, alliances, the sale of the company and stock repurchase.
Private investment firm Blue Harbour Group, which owns four per cent of Novell, approved the decision.
"We agree with Novell's management and board of directors that the company's value significantly exceeds Elliott's proposal. We support the company's decision today to pursue a formal review process with the assistance of its advisors," said the firm.
Engineer calculates that Chengdu's plan to replace streetlights with artificial moonlight would cost $100bn
Dark matter holds the Universe together - and gravitational waves could help identify it
Addison Lee is working on autonomous taxis for commuting and pleasure
IBM and Technical University of Munich team demonstrate how Shor's algorithm, which can't be cracked by conventional computers, can be solved quickly with quantum computing