add this line to styles section of angular.json:

"node_modules/quill/dist/quill.snow.css"

Firstly,put your component.ts

import { Editor } from 'primeng/editor';
declare var Quill: any;

and add this line in constructor

var fontSizeStyle = Quill.import('attributors/style/size');
fontSizeStyle.whitelist = ['24px', '48px', '100px', '200px'];
Quill.register(fontSizeStyle, true);

At the last,change your html

<span class="ql-formats">
   <select class="ql-size">
       <option value="24px">24</option>
       <option value="48px">48</option>
       <option value="100px">100</option>
       <option value="200px">200</option>
   </select>
</span>

Example

I have completely tested and migrated one of my existing project from Angular 5 to Angular 8. Applies to Primeng 5 to Primeng 8 migration. Carefully analysed what is happening in your case. Below is the only thing which you are missing out

In package.json:

    "primeicons": "^2.0.0",
    "primeng": "^8.0.3",
    "quill": "^1.3.7",
    "rxjs": "~6.3.3",

Note: Quill JS version should be 1.3 and above because you have migrated all other libraries.

And the expected output:

For your convenience I have also created the stackblitz with all your above code:

App URL : https://angular8-primeng-qull-editor.stackblitz.io

Editor URL: https://stackblitz.com/edit/angular8-primeng-qull-editor

To make it work:

Step 1: Update the package.json as mentioned above

Step 2: Do the npm install

Step 3: Serve it using ng s

Hope it should definitely work.