Experiences that are retrieved from memory are often not accurate, but prone to biases. The interplay of already existing semantic knowledge and recently generated episodic memory traces might explain some of the underlying mechanisms. The scenario construction model postulates that during encoding, only the gist of an episode is stored in the episodic memory trace and during retrieval, any needed information that is missing from that trace is constructed from semantic knowledge. The current study aimed to investigate semantic construction in a realistic, yet controlled setting. Using a desktop virtual reality (VR) participants navigated through a flat in which some household items appeared in unexpected rooms, creating conflicts between the experienced episode and semantic expectations. During recall after two separate retrieval delays (one day vs one week) we were able to identify influences from semantic knowledge. To manipulate salience, some objects were task-relevant, and some were irrelevant to the sequence of actions. We used spatial and temporal recall measures. Both congruency and task-relevance, but not time, predicted correct episodic memory retrieval. In the spatial memory retrieval, semantic construction was more likely than guessing in cases of episodic memory failure and occurred more frequently for task-irrelevant objects. In the temporal recall at the second retrieval delay we could show that object-pairs belonging to the same semantic room-category were temporally clustered together compared to object-pairs from different semantic room-categories. Taken together, our findings support the predictions of the scenario construction model, as we found influences of semantic knowledge on both spatial and temporal memory recall. The new VR-paradigm appears to be a promising tool for investigating semantic construction.