We analyze theoretically a hybridization type dressing of orbital (i.e., charge) and spin degrees of freedom of excitations captured in semiconductor quantum dot with band phonons and magnons (the latter for magnetic semiconductor surroundings), as a mechanism of dephasing of rapidly excited dot exciton state. Within the Green function approach we derive our previously formulated heuristic general rule for estimation of corresponding dephasing time-rate. The pure dephasing (off-diagonal decoherence) of quantum dot exciton resulting due to this dressing is studied, for both orbital and spin states of the exciton. A significant difference between phonon-induced (for charge) and magnon-assisted (for spin) quantum dot exciton dephasing is indicated as the result of the spin conservation, which leads to the disappearance of the exciton spin pure-dephasing at T = 0 (freezing out of spin dephasing), in contrary to the exciton charge dephasing caused by phonons, which maintains strong even at T = 0.