Computing with words (CWW) is an intelligent computing methodology for processing words, linguistic variables, and their semantics, which mimics the natural-language-based reasoning mechanisms of human beings in soft computing, semantic computing, and cognitive computing. The central objects in CWW techniques are words and linguistic variables, which may be formally modeled by abstract concepts that are a basic cognitive unit to identify and model a concrete entity in the real world and an abstract object in the perceived world. Therefore, concepts are the most fundamental linguistic entities that carries certain meanings in expression, thinking, reasoning, and system modeling, which may be formally modeled as an abstract and dynamic mathematical structure in denotational mathematics. This paper presents a formal theory for concept and knowledge manipulations in CWW known as concept algebra. The mathematical models of abstract and concrete concepts are developed based on the object-attribute-relation (OAR) theory. The formal methodology for manipulating knowledge as a concept network is described. Case studies demonstrate that concept algebra provides a generic and formal knowledge manipulation means, which is capable of dealing with complex knowledge and their algebraic operations in CWW.