In order to deliver real-time feedback to support employee development and rapid innovation, many companies are replacing formal review-based performance management with systems that enable frequent and continuous employee evaluation. Real-time feedback applications enable supervisors and employees to give, seek, and receive competency-based feedback using their computers, smartphones, or other devices. In this study, we examine the role of one such real-time feedback application to understand its effects on employee performance appraisals. First, we seek better understanding of how workplace relationships affect employee feedback across managers, colleagues, and direct reports and find that feedback tends to be more critical when given by managers. What is even more important from an industry perspective is the role of preferential treatment and retaliation. Managers can be more transactional, but colleagues are not. We also highlight a series of gender observations: men rate women higher than men, and women rate men and women similar to how men rate men. We conclude by finding that positive real-time feedback has a stronger effect on an employee’s future ratings than negative feedback. Our findings have direct implications for the design and implementation of performance management systems and highlight how companies can use information systems to create an innovative human resource operation that delivers flexibility and agility.