A flyback converter in general has three operating modes. They are continuous conduction mode (CCM), discontinuous conduction mode (DCM) and Transition mode (TM). Each mode has pros and cons. Quasi resonant flyback operating modes are best described using the current waveforms.
Continuous Conduction Mode (CCM)
In CCM, the energy stored in the primary is not collapsing to zero when the next switching cycle occur. The shapes of the currents will look like below figure. As you can see, the currents have step before it ramps up or downs linearly. This is a sign of a CCM operation.
The advantage of CCM is that it has lower ripple current. However, it needs also a bigger inductance so need to select a bigger transformer. With respect to loop control and compensation, CCM is difficult and complex.
Discontinuous Conduction Mode (DCM)
In DCM, the energy in the primary collapsed to zero before the next switching cycle occur. Refer to below figure. The region bounded in blue dotted lines shows the dead time. This is an indicator of operation in DCM.
The advantage of DCM is a smaller inductance and simple to deal with in terms of control system (this what control system experts said). However, the disadvantage is it has high peak current as so the ripple current seen by the output capacitor. This disadvantage is can be addressed by selecting properly the components though.
Transition Mode (TM)
The third quasi resonant flyback operating mode is called transition mode. It is also called a boundary mode sometimes. In this operating mode, the current on the secondary drops to zero in every start of a new switching cycle. There is no step on the current waveform nor a deadtime. Refer to below figure. In most analysis, the derivation of equations is being referred into this operating mode for easier analysis.
The inductance is less than the CCM but greater than DCM. The peak and ripple current are somewhere in between the CCM and DCM. Most of the times, design engineers use this mode in the analysis and equation derivation as this is the easiest to deal with among the rest.
Each of the quasi resonant flyback operating modes discussed above has its strength and weaknesses. It is up to the designer to which mode to select. In my personal experience, I already used both the DCM and TM operating modes. Basically, a quasi resonant flyback will operate in DCM at light load and approaches to TM at very heavy load. During the design stage, the designer will set the operating point in first valley switching for heavier loads for optimum performance. Read How Quasi Resonant Flyback Works – Detailed Operation to know further about valley switching.